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ABSTRACT 

Purposes and activities of the Targeted Research and 
Development Project No, 3 are described. The project seeks to 
identify and describe the national reading problem, methods of 
reading instruction, and the training of reading teachers. The 
project, a literature search, is designed to identify literature 
related to reading instruction, to develop a reliable and valid 
method for evaluating that literature, and to apply the method to the 
comprehensive body ot identified research. Steps 3n the project 
include (1) a search of the literature in the ERIC system, Education 
Index, Psychological Abstracts, Sociological Abstracts, Dissertation 
Abstracts, government publications, and a number of other sources, 
expected to identify approximately 1,200 reports; (2) a review of the 
documents by a preliminary selection committee; (3) a critical review 
by a second committee of qualified and trained readers; and (4) a 
cross validation of the literature to provide analytical summaries of 
aspects of reading for use by various audiences. Forms used in 
identifying and analyzing documents are included. (MS) 
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IITION OR POLICY, 



A Critical Review of the Information 
Base for Current Assumptions Regarding 
the Status of Instruction and Achlevement 
In Reading in the United States* 



Reginald Corder 
Advisory Services Department 
Educational Testing Service 
Berkeley, California 



This somewhat formidable title embraces the work being done 
in the Targeted Research and Development Project No. 3. Project 
3 is a literature search. It begins, as do Projects 1 and 2 V with 
the basic assumption that there exists a significant, quantifiable 
deficit between the present status of reading ichievement in the 
U.S. and the individual and social literacy needs of the populace. 

Project 3 focuses on three sub-areas of the literature about 
reading and addresses itself to three principal tasks: 

1. The determination of the extent and distribution of the 
national reading problem, 

2. The determination of the use frequency and use distribution 
of the instructional methods, approaches, materials and 
equipment for reading instruction. 

3. The description of the nature and extent of current practice 
in the training of those who teach children to read. 

★Prepared" for a^ymr^sltna, "Targeted Research and Development Program 
in Reading, Phase 1, Right to Read Effort, USOE National Center 
for Educational Research and Development", held on February S, 1971, 
at the Annual Meeting of the American Educational Research Association, 
in New York. 
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In addressing the several questions of this project, a central 
effort has been not only to discover the research and survey literature 
which bear upon the questions, but to develop and utilize a comprehensive, 
reliable and validated method for evaluating the quality of the 
literature. We have thus been involved in the design of a systematic 
procedure by which an analysis of the variety of surveys, test data 
and scientific literature can be constructed to determine whether 
adequate documentation of the present status of reading achievement, 
use of methods and materials, and the nature of teacher training is 
possible. 

In analyzing the total .requirements of Project 3, we have 
separatee the responsibilities and assigned them to several inter- 
related units: a project direction group; a bibliography development 
group which we have called the Logic Committee; a group of readers; 
and a report writing ejroup whveh we have called the Technical Review 
Committee . 

Project direction has, in addition to the obvious administrative 
responsibilities, the responsibility for developing comprehensive 
preliminary lists of the literature which relates to the three 
tasks of the study. The search list of documents is varied and 
extensive. Considerable effort has been made to be as comprehensive 
as possible and to utilize a rather broad definition of literature 
in an attempt to find the full range of appropriate documents. 

The preliminary search has involved a computer sweep through 
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all the ERIC documents, a hand search of the last ten years of the 
Education Index, Psychological Abstracts, Sociological Abstracts, 
Dissertation Abstracts, and of the publications of government agencies, 
state departments of education, selected educational research 
institutes and individuals, foundations, publishers of both reading 
materials and tests, teacher training institutions, and selected 
local school districts. 

The final selection of bibliographic materials for review 
involves a two-step process. First, all references developed 
through the preliminary selection procedure are submitted to an 
advisory group called the Logic Committee, persons with acknowledged 
expertneas in the fields of reading instruction, linguistics, and 
manpower resources. Members of this committee evaluate the preliminary 
set of references according to what they believe about the quality 
of the data so far as representativeness is concerned and the 
probable relevance of the references to one or more of the specific 
research tasks. These evaluations are then combined and a consensus 
as to inclusion is reached. 

Secondly, the references on which there is agreement are 
sorted out accoxflng to the particular sub-task to which they are 
relevant. A review and analysis of these references is then made 
through a series of matrices which provide explicit definitions of 
the parameters of the three tasks (the reading problem, methods of 
Instruction, and the training of teachers of reading) . A primary 

final body of literature to be reviewed is 
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to attempt to fill out the full matrix of population characteristics 
and environments for each task as these are described in the various 
surveys, documents and professional literature. Thus, although 
prolific documents or surveys might be available on selected population 
characteristics and selected learner environments, the primary 
purpose of choosing literature to fill out a complete matrix will 
take precedence over the review of all documents available on any 
one cell of a matrix* If it becomes apparent that the preliminary 
search has not revealed any references for certain cells, the Logic 
Committee is asked to help find references to fill these gaps. 

The general policy for accepting a document for review is that 
no reference is included in the bibliography without having been 
evaluated and ranked by all members of the Logic Committee, although, 
clearly, some references are included on which there is not total 
agreement as to worth or relevance* In following these procedures 
we hope to obtain a relatively unbiased body of documents to form 
the data base for the project. 

Actual critical review of all documents selected by the Logic 
Committee is performed by a Committee of Readers. These are doctoral 
candidates in reading, educational psyche logy, and sociology from 
the University of California, Berkeley. In their review the readers 
use a single review format which permits them to make judgements 
concerning the adequacy of the data reported, and (as it may be 
appropriate) the degree to which the reported characteristics, 
conceptualizations, methodology or status may be determined from 

fcKJC 
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the evidence presented in the literature. The review format itself 
was developed by members of the ETS research staff in the Western 
Office who will be responsible for writing the final state-of-the-art 
reports. This group, which we have called the Technical Review 
Committee, has also been responsible for training the readers in the 
use of the review format. Information included in the format is 
both objective and subjective, as in addition to reviewing a document 
for specific population or environmental characteristics which are 
relatively easy to define, readers are asked to make judgements 
concerning data quality and representativeness according to the 
Gephart model. Reviews also include relevant data tables And an 
abstract . 

Initially, members of both the Technical Review Committee 
and the Logic Committee expressed concern about the writing of state- 
of-the-arL papers based upon reviews piepared by graduate student 
readers. After a period of training and trial review of a number 
of documents by the readers, members of both committees became 
satisfied with the work of these reviewers, however. Every fourth 
document approved by the Logic Committee is reviewed independently 
by two readers as a reliability check. Members of the project 
direction group and Technical Review Committee also review some 
documents independently, as an Additional check on quality control. 

The final aspect of the rationale for review is a procedure 
by which cross-validation or verification of the data acquired from 

O 

ERIC 
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the literature search may be made. Following the complete review 
of the documents finally selected, pilot investigations will be 
utilized to ascertain the accuracy of selected written reports 
dealing with the three tasks of this project. A munber of field 
studies of two days each will be made as additional validations of 
the crdibility of any large-scale reports which seem to represent 
significant trands or conclusions from the literature reviewed. 

In summary, we expect that the work done in Project 3 will: 

1, Identify approximately 1200 surveys, reports or studies 
which deal with the three sub-tasks of the project and are 
judged to be significantly important by an independent 
group of acknowledged experts in the fields of reading, 
linguistics, and manpower resources. 

2. Establish a reviewing format with explicit criteria for 
rating and analyzing research reports and other literature 
in the field of reading and demonstrate the training of 
readers in the reliable use of this format. 

3.. Provide critical reviews of the 1200 significant docuim ts 
by a group of trained and highly qualified readers. 

4. Present a summary of the significant literature regarding 
reading which has been analyzed ard classified by a known 
and consistently applied set of criteria. 
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5. Present a synthesis of the information extracted from the 
analysis of 1200 significant documents to provide at least 
partial answers to the questions pertaining to the teaching 
of reading and the reading status of the many sub-populations 
in the United States. 

6. Provide verification of the accuracy of the information 
obtained from the literature by site visitations to regions 
reporting reading status and procedures for the teaching 

of reading that appear to be trend-setting. 

7. Present the critical reviews, reports, and syntheses which 
are the outcomes of this project in a form immediately 
compatible with the ERIC system of retrieval. 
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Reject Accept 



I. Is this a test of an hypothesis? (experimental, quasi-experimeutal 
or or demonstration only) YES NO 

^ II. Is this an answer to an empirical question? (survey, description, 
exploratory question or review of literature) YES N O 

(Now Complete Search List Pages 2, 3, 4) 



III. (Answer to Question I is yes) The hypothesis is supported by 
the author's conclusion (s) ? 

Not Supported Somewhat Supported Supported 

IV. (Answer to Questions I or II is yes) Was the quality of the data 

generation procedure Appropriate Inappropriate N. A_ 

V. (Answer to Questions I or II is yes) Was the data analysis 

Appropriate Inappropriate N. A 

VI. This article describes the status of Task I Task II Task III 

VII. Comments: 
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ABSTRACT: Includes purpose, method, conclusion, and reviewer's critique 
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List relevant tables , annotate for clarity and Interpretation and attach. 
Data Tables included are: 



Aside from the adequacy of the research, are there any innovative or useful 
ideas contained in this study? (implications for future research, or ideas 
the author did not test?) Specify. 
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